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ABSTRACT 


Idaho uses the English Language Arts and Mathematics tests from the Smarter 
Balanced Assessment Consortium (SBAC) for the Idaho Standard Achievement 
Tests. ISAT results have been reported almost exclusively as "percent proficient 
or above" statistics (i.e., the percentage of Idaho students who performed at 
the "A" level). This paper illustrates how a traditional normative statistic using 
scale scores -- percentiles -- can provide informative descriptions of student 
achievement trends. The 10th, 25th, 50th, 75th and 9Oth percentiles for the 
ISAT 2015, 2016, and 2017 English Language Arts and Mathematics tests for 
grades 5-8 and 10 were examined. These five percentiles provided an across- 
the-board look at student achievement (i.e., High, High Average, Average for 
Grade Level, Low Average, and Low performing students) and at how 
achievement has changed over time and graphically via boxplots and line 
graphs. Between 2016 and 2017, Idaho lost 26 percentile points statewide in 
English language arts but gained three in mathematics. Statewide from 2016 to 
2017, the English language arts percentiles of Idaho’s High and High Average 
performing students increased while the percentiles of Idaho’s Average, Low 
Average, and Low performing students decreased. The mathematics percentiles 
of Idaho’s High, High Average, and Average performing students increased 
while the percentiles of Idaho’s Low Average and Low performing students 
decreased. These achievement trends in terms of percentiles could become an 
informative element of Idaho’s comprehensive accountability system. 
(Contains 7 references, 2 tables, and 16 figures. ) 
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Learning Point Associates (2009) has suggested three measures of school 
performance that can provide valuable support for program evaluation and 
accountability systems. The three measures are averages, percentiles, and 
effect sizes. This paper considers what we might learn about the statewide 
performance of Idaho students from their percentile results on the 2015, 2016 
and 2017 ISAT (SBAC) tests of English language arts and mathematics. 


Average scores are amenable to traditional statistical analysis, but they hide a 
lot of performance information in a single measure of central tendency. 
Percentiles are much less amenable to statistical analysis, but they provide 
performance information across the distribution of scale scores. They enable 
us to look at the change in five performance levels from year to year, and five 
overall trends. The five percentiles (i.e., scale scores) are: 


90th percentile: High Score, above 90% of the scale scores on the test 

75th percentile: High Average Score, above 75% of the scale scores 

50th percentile: Average Score for Grade Level, above 50% of the scale scores 
25th percentile: Low Average Score, above 25% of the scale scores 

10th percentile: Low Score, above 10% of the scale scores 


The 75th and 25th percentiles define the middle half of students who took the 
test, and that the 50th percentile is the middle half of students and the middle 
of all students. 


DATA COLLECTION 


The ISAT by SBAC measured student performance in English language arts and 
mathematics in grades 5 through 8 and 10, and reported out the results as 
four-digit scale scores. Thus, the first task was to identify 210 percentiles (i.e., 
the five percentiles x the seven grades tested x the two subjects tested x the 
three ISAT administrations). 


The State of Idaho (i.e., the State Department of Education or the State Board 
of Education) does not report out ISAT percentile results. The percentiles were 
collected from three ISAT scale score to percentile rank conversion tables that 
are available on the web (Stoneberg, 2015; Stoneberg, 2016a; Stoneberg, 
2017). These conversion tables were prepared using Microsoft Excel to process 
ISAT 2015, 2016 and 2017 summary statistics provided by the Idaho State 
Department of Education. 
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The 140 ISAT percentiles (i.e., scale scores) required to conduct the analyses 
reported in this paper are listed in Table 1. 


Table 1. Idaho scale scores at the 10th, 25th, 50th, 75th and 90th percentiles for the 
ISAT/SBAC English language arts and mathematics tests from 2015, 2016, and 2017 in 
grades 3-8 & 10. 


ISAT English Language Ats, Percentiles 2015 to 2047, Grades 3-8 and 10 
ens) ane] a7 aut [an [ani [ants [ani [an 


Oth] 2324 2323 2315 ©2352 2356 2350 2395 2382 2386 216 2386 2413 2433 2436 2433 2452 2452 269 469 2470 2458 
ith] 2572 2573 2368 DAD 2409 244 24M5 2445 2442 2467 2855 2467 2MB7 2691 2489 2506 2506 2505 2590 2531 2522 
Sth] 2426 2428 2421 24Gt 2468 2463 2502 2506 2504 2504 2527 2525 O48 2552 261 2566 2586 2667 2590 2500 2597 
Pith] 2479 2483 2475 2549 2527 2523 250) 2506 2567 2581 2573 2586 2008 2614 2614 2628 2626 2690 2086 2667 268) 
Q0th| 2528 2533 2524 2570 2500 2577 2611 2600 2623 2632 2618 240 2062 2069 2670 2085 2680 2600 227 28 D7 


ISAT Mathematics, Percentiles 2015 to 047, Grades 3-8 and 40 
aut [ani [an aut ani [an aut [ani [an 


Oth] 2337 2339 2319 2375 2378 2374 2385 2395 2302 2387 2402 20h D0 DG 2G DNH2 2414 2M DAS DANG DAA 
ith] 2382 2385 2368 D421 2425 2419 2445 2446 2445 2053 2059 2462 2468 2476 247 2476 2479 2476 2482 2483 2481 
Sth] 2432 2896 2433 471 2478 2475 2500 2503 2504 2516 2522 2532 2533 2542 2542 2547 25562 2551 2555 2558 2550 
Pith] 2482 2487 2405 2522 2526 2529 2555 2559 2564 2579 2505 2000 2598 2008 2008 2617 2624 2627 2629 2632 268 


Q0th| 2527 2533 254 2567 2578 2577 2605 2610 262 2696 2641 2060 2057 2067 267 2081 29) 2604 2085 2706 7708 


RESULTS: GRADE LEVEL BY PERCENTILE TABLES 


One way to look at Idaho’s percentile results is to identify the point differences 
between the 2016 and 2017 assessments for each grade-level and each 
percentile and prepare a seven-grade by five-percentile table (with summary 
point counts for each grade-levels and each percentile completing an 8 by 6 
table). 


The 2016 to 2017 point differences for the ISAT English language arts 
mathematics tests are in Table 2. The author suggests only a few narratives 
related for Table 2. The intent is that the reader will take the opportunity to 
review the data and to write personal narratives for findings of specific 
interest to the reader. The reader may also want to mine the information in 
the three-year boxplots and line graphs in Appendix A for further personal 
narratives. 
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Table 2. Summary changes by grade level and percentile from 2016 to 2017 on the 
ISAT/SBAC English language arts and mathematics tests. 


Idaho Percentile Changes, 2016 to 2017, Grades 3-8 and 10 


ISAT/SBAC English Language Arts 
CE 
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erases | 3) Be = | | 
2 4 
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5 -34 
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7 
0 
0 
0 
0 
0 


9 
5 
1 
1 
-3 


Percentile changes in ISAT English language arts, 2016-2017 


1) Of the 35 grade-percentile changes on the 2017 English Language Arts test, 
11 were positive, one was zero, and 26 were negative. 

2) Sixth graders had five of the eight “grade-level” gains in 2017. 

6) Sixth graders with a total of 85 points across the five percentiles (i.e., levels 
of performance) had the most improved performance on the English 
Language Arts test in 2017. 

4) The overall statewide results for ISAT English Language Arts: Idaho scored 
26 points lower in 2017 than in 2016. 


Percentile changes ISAT Mathematics, 2016-2017 


1) Of the 35 grade-percentile changes on the 2017 ISAT Mathematics test, 15 
were positive, five were the zero, and 15 were negative. 

2) Sixth graders with a total gain of 50 points across the five percentiles had 
the most improved statewide performance in Mathematics in 2017. 

5) Seventh grade had the same percentiles in 2017 as in 2016 in Mathematics. 

4) The overall statewide results for ISAT English Language Arts: Idaho scored 
6 points higher in 2017 than in 2016. 
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RESULTS: 2016-2017 CHANGE BY GRADE AND BY PERCENTILE 


Change from 2016 to 2017 by Grade 


Percentile Points 
o 


Figure 1. Total change in percentile points from 2016 to 2017 across the High (90th), 
High Average (75th), Average (50th), Low Average (25th) and Low (10th) percentiles for 
grades 3, 4, 5, 6, 7, 8 and 10, for ISAT English language arts and mathematics. 


English language arts, 2016-2017 

Statewide from 2016 to 2017, Idaho students in two grades (6th and 8th) 
gained points in English language arts, while students in five grades (drd, 4th, 
5th, 7th, and 10th) lost points. 

Mathematics, 2016- 2017 

Statewide from 2016 to 2017, Idaho students in three grades (5th, 6th and 


10th) gained points in mathematics, students in three grades (Srd, 4th, and 
8th) lost points, and 7th grade students maintained their status quo. 
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Change from 2016 to 2017 by Percentile 


50 


25 


Percentile Points 


Figure 2. Total change from 2016 to 2017 across grades 3, 4, 5, 6, 7, 8 and 10 for the 
High (90th), High Average (75th), Average (50th), Low Average (25th) and Low (10th) 
percentiles, for English language arts and mathematics. 


English Language Arts, 2016-2017 

Statewide from 2016 to 2017, the English language arts percentiles of Idaho’s 
High and High Average performing students increased while the percentiles of 
Idaho’s Average, Low Average, and Low performing students decreased. 
Mathematics, 2016-2017 

Statewide from 2016 to 2017, the mathematics percentiles of Idaho’s High, 


High Average, and Average performing students increased while the 
percentiles of Idaho’s Low Average and Low performing students decreased. 
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RESULTS: BOXPLOTS AND LINE GRAPHS 


Performance trends being with the third data point and are refined with each 
additional data point. The author selected two kinds of graphs for displaying 
trends at the 10th, 25th, 50th, 75th and 9Oth percentiles over time for each 
grade-subject: boxplots and line charts. Line charts are often used to report 
student test results and are generally understood by administrators, teachers 
and parents. Boxplots, however, are not commonly used to report student test 
results so a brief overview is in order. 


Overview of Boxplot: ISAT Percentiles 


2600 


90th percentile (90% of scores below this value) 
High Score 


——— 75th vercentile (75% of scores below this value) 
High Average Score 


80% of 50% of 
scores are scores 
between are in 
the tails the box 


50th percentile (50% of scores below this value) 
Average for Grade Level 


25th percentile (25% of scores below this value) 
Low Average Score 


10th percentile (10% of scores below this value) 
Low score 


i} 


The boxplot permits the reader to follow the academic progress of the middle 
half of students (i.e., from high average and average for grade level to low 
average) without losing track of the high and low students. While it does not 
provide the reader with exact scores for the five percentiles, the boxplot does 
illustrate a trend. The reader, however, will find exact scale scores for the 
high, high average, average, low average, and low percentiles displayed in the 
line graphs. 


The full-page figure for each grade-subject (e.g., third grade English Language 
Arts, etc.) in Appendix A displays both the boxplot and line graph for the 
statewide ISAT 2015, 2016, and 2017 High, High Average, Average, Low 
Average and Low percentiles. 


1) English language arts. Boxplot and line graph displays for grades 3-8 and 
10 percentile results are in Figures Al through A’ in Appendix A. 


2) Mathematics. Boxplot and line graph displays for grades 3-8 and 10 
percentile results are in Figures A8 through Al4 in Appendix A. 
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DISCUSSION 


Percent Proficient vs. Scale Scores. Each year Idaho spends a considerable 
sum to administer and score the Idaho Standards Achievement Tests. A search 
for ISAT results on the State Department of Education and the State Board of 
Education website yielded limited information about the test itself or the 
administrative logistics that school districts must follow, but there were little 
to no ISAT English Language Arts and Mathematics test results other than 
“percent below basic, basic, proficient, advanced” posted. See, for example, 
http://www.sde.idaho.gov/communications/files/frequently-requested-data/2016- 
2017-Assessment-Results.xlsx 


The percent above cut-score metric such as those reported by ISAT and NAEP 
are problematic, especially when looking at achievement trends. The National 
Academies of Sciences, Engineering, and Medicine (2017) conducted a 
congressionally mandated evaluation of the NAEP achievement levels — Basic, 
Proficient and Advanced (i.e., NAEP’s percent above cut-score metric) — to 
determine whether the achievement levels are reasonable, reliable, valid, and 
informative to the public, and to recommend ways that the use of achievement 
levels can be improved. The National Academies report notes: 


One of the most common and unwarranted inferences using 
achievement level percentages involves assessing the amount of 
progress students have made over time, particularly by population 
groups. For instance, news reports often focus not only on how 
students are doing at a particular time, but the extent to which the 
percentage of students scoring Proficient or above has (or has not) 
improved over successive NAEP years. When these comparisons 
are based on the scale scores, they provide useful information. 
When they are based on the “percentage Proficient or above” 
metric and used to compare progress across groups, they can be 
misleading. A report by Holland on this issue focused on 
misinterpretations associated with using the “percent above a cut 
score” metric. Although this metric is widely used for NAEP [and 
ISAT], there are serious limitations to the inferences that the 
percent above cut score metric can support, particularly when 
evaluating trends over time, gaps among groups, or trends in 
gaps.... [Emphasis added.] (p. 208). 


The percentile results reported in this paper were based on ISAT 2015 through 
2017 scale scores. Percentiles provide useful trend information that is 


preferred over percent above cut-score trend information. 


# # # 
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APPENDIX A 


ISAT English Language Arts 
2015, 2016, and 2017 


Grade-level boxplot and line graphs for statewide scale scores at the 
10th, 25th, 50th, 75th, and 9Oth percentiles 


Grade 3, Page 11 
Grade 4, Page 12 
Grade 5, Page 13 
Grade 6, Page 14 
Grade 7, Page 15 
Grade 8, Page 16 
Grade 10, Page 17 


ISAT Mathematics 
2015, 2016, and 2017 


Grade-level boxplot and line graphs for statewide scale scores at the 
10th, 25th, 50th, 75th, and 9Oth percentiles 


Grade 3, Page 18 
Grade 4, Page 19 
Grade 5, Page 20 
Grade 6, Page 21 
Grade 7, Page 22 
Grade 8, Page 23 
Grade 10, Page 24 
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English Language Arts: ISAT 2015-2020 
: Idaho, Grade 3, Percentile Boxplots 
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Figure A1. ISAT English Language Arts Test, Grade 3, 2015 to 2017. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 
formats. 
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English Language Arts: ISAT 2015-2020 
: Idaho, Grade 4, Percentile Boxplots 
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Figure A2. ISAT Reading Language Arts Test, Grade 4, 2015 to 2017. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 
formats. 
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English Language Arts: ISAT 2015-2020 
: Idaho, Grade 5, Percentile Boxplots 
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Figure A3. ISAT English Language Arts Test, Grade 5, 2015 and 2016. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 
formats. 
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English Language Arts: ISAT 2015-2020 


- Idaho, Grade 6, Percentile Boxplots 
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English Language Arts: ISAT 2015-2020 


Scale Scores by Percentile 


Idaho, Grade 6 
2632 _— 
2618 
2586 
foe ee 
2524 2527 2526 - $-10th 
ios iil eran © -@-25th 
2467 2467 ~"@--50th 
en co = oO —a-—75th 
2416 2413 —e 90th 
x. ~_ 2386 _-® 
~ > - 
2015 2016 2017 2018 2019 2020 


Figure A4. ISAT English Language Arts Test, Grade 6, 2015 to 2017. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 
formats. 


Visit http://k12researchidaho.com Page 14 of 24 


Idaho's SBAC Percentile Results 2017 


English Language Arts: ISAT 2015-2020 


: Idaho, Grade 7, Percentile Boxplots 
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Figure A5. ISAT English Language Arts Test, Grade 7, 2015 and 2016. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 


formats. 
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English Language Arts: ISAT 2015-2020 


: Idaho, Grade 8, Percentile Boxplots 
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Figure A6. ISAT English Language Arts Test, Grade 8, 2015 to 2017. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 
formats. 
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Figure A7. ISAT English Language Arts Test, Grade 10, 2015 to 2017. Idaho statewide scale 
scores for the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph 
formats. 


Visit http://k12researchidaho.com 


Page 17 of 24 


Idaho's SBAC Percentile Results 2017 Revised 01/17/2018 


Mathematics: ISAT 2015-2020 
Idaho, Grade 3, Percentile Boxplot 
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Figure A8. ISAT Mathematics Test, Grade 3, 2015 to 2017. Idaho statewide scale scores for the 
10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 


Visit http://k12researchidaho.com Page 18 of 24 


Idaho's SBAC Percentile Results 2017 Revised 01/17/2018 


Mathematics: ISAT 2015-2020, 
Idaho, Grade 4, Percentile Boxplot 
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Figure AQ. ISAT Mathematics Test, Grade 4, 2015 to 2017. Idaho statewide scale scores for the 
10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 
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Mathematics: ISAT 2015-2020 
Idaho, Grade 5, Percentile Boxplot 
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Figure A10. ISAT Mathematics Test, Grade 5, 2015 to 2017. Idaho statewide scale scores for 
the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 
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Mathematics: ISAT 2015-2020 
Idaho, Grade 6, Percentile Boxplot 


2675 
2650 
2625 
2600 
2575 
2550 
2525 
2500 
2475 
2450 
2425 
2400 
2375 


2015 2016 2017 2018 2019 2020 


Mathematics: ISAT 2015-2020 
Scale Scores by Percentile 
2660 Idaho, Grade 6 


2600 
-¢-10th 
2532 
2516 ao22 ke -@-25th 
@--------- -— 
--@--50th 
2453 2459 2462 —#—75th 
G@----- a----- a —e 90th 
2397 2402 2405 
@----- eo 2 
2015 2016 2017 2018 2019 2020 


Figure A11. ISAT Mathematics Test, Grade 6, 2015 to 2017. Idaho statewide scale scores for 
the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 
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Mathematics: ISAT 2015-2020 
Idaho, Grade 7, Percentile Boxplot 
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Figure A12. ISAT Mathematics Test, Grade 7, 2015 to 2017. Idaho statewide scale scores for 
the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 
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Figure A13. ISAT Mathematics Test, Grade 8, 2015 to 2017. Idaho statewide scale scores for 
the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 
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Figure A14. ISAT Mathematics Test, Grade 10, 2015 to 2017. Idaho statewide scale scores for 
the 10th, 25th, 50th, 75th, and 90th percentiles presented in boxplot and line graph formats. 
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